Skip to content

Add a --min-time option to make small benchmarks run more iterations#1324

Open
lialan wants to merge 6 commits into
mainfrom
users/lialan/min_time
Open

Add a --min-time option to make small benchmarks run more iterations#1324
lialan wants to merge 6 commits into
mainfrom
users/lialan/min_time

Conversation

@lialan

@lialan lialan commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

Previously we've seen measured time swings of the benchmark results:

  • can swing as much as 15%
  • usually makes the measuring of small benchmarks inaccurate.

This patch adds a --min-time option, which will:

  • first measure a single warmup run.
  • calculate how many runs can saturate X seconds specified by min-time.
  • if --iter value is smaller than the calculated value, replace it with the calculated value.
  • this applies to each benchmark entry separately, so each bench gets different iterations.

To give an example, with --min-time 5, a non-trivial gemm:

aten::mm '[[4096, 8192], [8192, 2048]]'

will gets auto-adjust:

Auto-adjusted iterations: 6867 (warmup: 0.0007s, target: 5.0s, floor: 100)

which should amortize away a lot of the swings. Since 0.0007s is still a bit noisy.

@rkayaith

rkayaith commented Mar 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Thanks I think this is a great idea! I haven't looked at the code in detail yet, so just some thoughts here on the interface.
Reusing --iter as a minimum iteration count is a bit confusing, as so far it has meant an exact number of iterations, and the naming isn't clear. It also seems inconvenient to require --min-time 0 to get the old behaviour. I would suggest:

  • adding a new flag like --min-iter (could be a different name) which we default to 100
  • change the behaviour of --iter so that --iter X just maps to --min-iter X --min-time 0 internally. it should be mutually exclusive with those flags

The default behaviour when no flags are used should be the same as you proposed here, but with this any existing usage of --iter won't need to be changed.

@lialan

lialan commented Mar 11, 2026

Copy link
Copy Markdown
Contributor Author

Thanks I think this is a great idea! I haven't looked at the code in detail yet, so just some thoughts here on the interface. Reusing --iter as a minimum iteration count is a bit confusing, as so far it has meant an exact number of iterations, and the naming isn't clear. It also seems inconvenient to require --min-time 0 to get the old behaviour. I would suggest:

  • adding a new flag like --min-iter (could be a different name) which we default to 100
  • change the behaviour of --iter so that --iter X just maps to --min-iter X --min-time 0 internally. it should be mutually exclusive with those flags

The default behaviour when no flags are used should be the same as you proposed here, but with this any existing usage of --iter won't need to be changed.

What about just by default set --min-time to 0, and then change --iter to --min-iter, it basically does the same thing like you suggested.

@rkayaith

rkayaith commented Mar 12, 2026

Copy link
Copy Markdown
Member

Having the default automatically run for some set amount of time seems useful. I took a look at some other benchmarking tools, and it seems in line with what they do as well.

@lialan

lialan commented Mar 12, 2026

Copy link
Copy Markdown
Contributor Author

Having the default automatically run for some set amount of time seems useful. I took a look at some other benchmarking tools, and it seems in line with what they do as well.

Those changes are made:

  • --iter X now means "run exactly X iterations" (disables auto-adjustment). Internally maps to --min-iter X --min-time 0.
  • --min-iter (default 100): iteration count used when auto-timing is disabled.
  • Backward compatible: existing --iter 100 usage keeps working (exact 100 iterations, same as before).

Turn on min-iter when you need.

@rkayaith

lialan and others added 6 commits March 30, 2026 08:44
@lialan @claude
[BOO] Auto-adjust benchmark iterations based on minimum time target
d743080 
Add --min-time flag (default 3.0s) to ensure benchmarks run long enough
for stable timing. Times the warmup call, computes needed iterations via
ceil(min_time / warmup_time), and uses max(computed, --iter) as the
actual count. --iter becomes a floor rather than an exact count.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lialan @claude
[BOO] Make --min-time take priority over --iter
39b7400 
Previously --iter acted as a floor, so --min-time could only increase
iterations beyond --iter. Now when --min-time is active, its computed
iteration count is used directly. --iter is only a fallback when
--min-time is disabled (set to 0).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lialan
Fix warm up run time measurement.
41db2bd
@lialan
Address linting issue.
1b6616d
@lialan
Update comments.
f19ca76
@lialan
Add missing parts.
618e3d4
@lialan lialan force-pushed the users/lialan/min_time branch from 46d81fe to 618e3d4 Compare March 30, 2026 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rkayaith rkayaith Awaiting requested review from rkayaith rkayaith is a code owner

@zjgarvey zjgarvey Awaiting requested review from zjgarvey zjgarvey is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lialan @rkayaith